Image processing device, endoscope system, and image processing method

ABSTRACT

There is provided an image processing device comprising a processor, in which the processor acquires a plurality of endoscopic images obtained by picking up images of an observation target with an endoscope, calculates a raw score related to a determination of a severity or a stage of a disease of the observation target, on the basis of each of the endoscopic images, decides a final score on the basis of the raw score, and performs control to display the final score and/or a change over time of the final score in real time on a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2021/002286 filed on 22 Jan. 2021, which claims priority under 35U.S.C § 119(a) to Japanese Patent Application No. 2020-019803 filed on 7Feb. 2020. The above application is hereby expressly incorporated byreference, in its entirety, into the present application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing device, anendoscope system, and an image processing method that perform diagnosissupport by using an endoscopic image captured by an endoscope.

2. Description of the Related Art

In a medical field, a diagnosis using an endoscope system comprising alight source device, an endoscope, and a processor device has beenwidely performed. In the diagnosis using the endoscope system, acomputer-aided diagnosis (CAD) technology for performing appropriateimage processing on an image (hereinafter, referred to as an endoscopicimage) obtained by imaging an observation target with the endoscope todetermine the stage or the like of a specific disease of the observationtarget has been developed. With the CAD used, for example, the severityof the disease, a score associated with a pathological result, or thelike is calculated in real time or near real time and is displayed on adisplay or the like.

Techniques for more convenient use of the CAD are disclosed. Forexample, there is known an image analysis device capable ofautomatically discriminating between a super-magnified image and anon-magnified image for CAD using a super-magnified image in anendoscopic image (JP2019-111040A).

SUMMARY OF THE INVENTION

In a real-time CAD using an endoscopic image, a stable determinationresult may not be obtained because the position of an observation targetin the endoscopic image is easily changed. For example, in a case ofdetermining the stage of ulcerative colitis by performing imageprocessing on the endoscopic image, the determination result of thestage may be not stable because the determination result is also finelychanged in a case where an observation position is finely changed.Therefore, it is desired to obtain a stable determination result inorder to make the CAD easier to use.

The present invention has been made in view of the above circumstances,and an object of the present invention is to provide an image processingdevice, an endoscope system, and an image processing method that stablydisplay a determination result regarding a disease by using anendoscopic image.

The present invention relates to an image processing device comprising aprocessor. The processor acquires a plurality of endoscopic imagesobtained by picking up images of an observation target at timesdifferent from each other with an endoscope, calculates a raw scorerelated to a determination of a severity or a stage of a disease of theobservation target, on the basis of each of the endoscopic images,decides a final score on the basis of the raw score, and performscontrol to display the final score and/or a change over time of thefinal score in real time on a display.

It is preferable that the processor calculates two or more types of rawscores different from each other.

It is preferable that the processor calculates the raw score on thebasis of a first feature amount obtained by analyzing the endoscopicimage.

It is preferable that the first feature amount is an amount related to asuperficial blood vessel dense part, an intramucosal hemorrhage part, oran extramucosal hemorrhage part included in the endoscopic image.

It is preferable that the processor executes a trained first machinelearning model generated by an input of a past endoscopic imageassociated with the raw score to a machine learning model, andcalculates the raw score on the basis of the endoscopic image.

It is preferable that the processor decides the final score from the rawscore that is calculated on the basis of a plurality of the endoscopicimages acquired in a predetermined period before a point in time atwhich the final score is decided.

It is preferable that the processor decides the final score byperforming a moving average, or FIR filtering or IIR filtering of aplurality of the raw scores.

It is preferable that the processor decides the final score on the basisof the raw score calculated immediately before or immediately after apoint in time at which the final score is decided.

It is preferable that the processor discriminates, for each endoscopicimage, whether the endoscopic image is suitable or unsuitable for thecalculation of the raw score, and defines the raw score to beuncalculated for the endoscopic image discriminated to be unsuitable forthe calculation of the raw score.

It is preferable that the processor discriminates whether the endoscopicimage is suitable or unsuitable for the calculation of the raw score, onthe basis of a second feature amount of the endoscopic image.

It is preferable that the second feature amount is an amount related toat least one selected from the group consisting of halationdistribution, spatial frequency distribution, brightness valuedistribution, shadow distribution, a magnification ratio indicator, andreflected light distribution of illumination light of the endoscopicimage, the illumination light being emitted to the observation target.

It is preferable that the processor executes a trained second machinelearning model generated by an input of a past endoscopic image to amachine learning model, the past endoscopic image being associated withwhether the endoscopic image is suitable or unsuitable for thecalculation of the raw score, and discriminates whether the endoscopicimage is suitable or unsuitable for the calculation of the raw score.

It is preferable that the processor decides the final score on the basisof the raw score except for the raw score to be uncalculated.

It is preferable that the processor defines the final score to beuncalculated in a case where the number of the raw scores except for theraw score to be uncalculated is a predetermined number or less, a casewhere the number of the raw scores to be uncalculated is a predeterminednumber or more, the raw scores being based on the plurality ofendoscopic images acquired in the predetermined period, or a case wherea ratio of the number of the raw scores to be uncalculated to the numberof the raw scores based on the plurality of endoscopic images acquiredin the predetermined period is a predetermined value or more.

It is preferable that the change over time of the final score isdisplayed by at least one graph showing a relationship between the finalscore and a decision time of the final score.

It is preferable that the processor determines a site of the observationtarget included in the endoscopic image by performing image analysis onthe endoscopic image, and the change over time of the final score isdisplayed by at least one graph showing a relationship between the finalscore, and a decision time of the final score and the site.

It is preferable that the processor gives an instruction to acquire astill image, and performs control to display the final score and/or thechange over time of the final score in a case where the instruction isgiven.

It is preferable that the disease is ulcerative colitis.

Further, the present invention relates to an endoscope systemcomprising: an endoscope that picks up an image of an observationtarget; and an image processing device provided with a processor. Theprocessor acquires a plurality of endoscopic images obtained by pickingup images of an observation target at times different from each other,calculates a raw score related to a determination of a severity or astage of a disease of the observation target, on the basis of each ofthe endoscopic images, decides a final score on the basis of the rawscore, and performs control to display the final score and/or a changeover time of the final score in real time on a display.

Further, the present invention relates to an image processing methodcomprising: an image acquisition step of acquiring a plurality ofendoscopic images obtained by picking up images of an observation targetat times different from each other; a raw score calculation step ofcalculating a raw score related to a determination of a severity or astage of a disease of the observation target, on the basis of each ofthe endoscopic images; a final score decision step of deciding a finalscore on the basis of the raw score; and a display control step ofperforming control to display the final score and/or a change over timeof the final score in real time on a display.

According to the present invention, it is possible to stably display adetermination result regarding a disease by using an endoscopic image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view of an endoscope system.

FIG. 2 is an external view of an operation part of an endoscope.

FIG. 3 is a block diagram showing a function of the endoscope system.

FIG. 4 is a diagram illustrating four-color LEDs provided in a lightsource unit.

FIG. 5 is a graph showing spectra of violet light V, blue light B, greenlight G, and red light R.

FIG. 6 is a graph showing spectra of special light.

FIG. 7 is a graph showing a spectrum of special light including onlyviolet light V.

FIG. 8 is a block diagram showing a function of a score processing unit.

FIG. 9 is a view illustrating a pattern of a blood vessel structure thatvaries depending on a severity of ulcerative colitis.

FIG. 10 is a schematic view schematically showing a cross-section of alarge intestine.

FIG. 11 is a diagram illustrating that a superficial blood vessel densepart, an intramucosal hemorrhage part, and an extramucosal hemorrhagepart are classified by brightness values and spatial frequencies.

FIG. 12 is a diagram illustrating relation between an endoscopic imageand raw score calculation in time series.

FIG. 13 is a diagram illustrating display control of a final scoreobtained by a moving average of raw scores.

FIG. 14 is a diagram illustrating display control of a final score inwhich a site name is displayed.

FIG. 15 is a diagram illustrating display control of a final score basedon the raw score obtained immediately before.

FIG. 16 is a diagram illustrating display control of a final score basedon the raw score obtained immediately after.

FIG. 17 is a diagram illustrating relation between the endoscopic imageand calculation of a raw score to be uncalculated in time series.

FIG. 18 is a diagram illustrating display control of a final score in acase where the raw score to be uncalculated is included.

FIG. 19 is a diagram illustrating display control of a final scoreshowing a discrimination result.

FIG. 20 is a flowchart showing a series of flows of a score displaymode.

FIG. 21 is a block diagram showing a diagnosis support device.

FIG. 22 is a block diagram showing a medical service support device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In FIG. 1 , an endoscope system 10 includes an endoscope 12, a lightsource device 14, a processor device 16, a display 18, and a console 19.The endoscope 12 is optically connected to the light source device 14and is electrically connected to the processor device 16. The endoscope12 includes an insertion part 12 a that is inserted into a body of anobservation target, an operation part 12 b that is provided at theproximal end portion of the insertion part 12 a, and a bendable portion12 c and a distal end portion 12 d that are provided in the insertionpart 12 a on the distal end side. An angle knob 12 e (see FIG. 2 ) ofthe operation part 12 b is operated so that the bendable portion 12 c isoperated to be bent. The bendable portion 12 c is operated to be bent sothat the distal end portion 12 d is made to face in a desired direction.

As shown in FIG. 2 , the operation part 12 b includes a mode changeoverswitch 12 g that is used to perform an observation mode switchingoperation, a zoom operation portion 12 h that is used to change an imagepickup magnification, and a still image acquisition instruction portion12 f through which a still image acquisition instruction is given, inaddition to the angle knob 12 e. An operation or an instruction using aconsole 19, a foot switch (not shown), or the like, in addition to themode changeover switch 12 g or a scope switch of the still imageacquisition instruction portion 12 f, may be used for the observationmode switching operation, the zoom operation, or the still imageacquisition instruction.

The endoscope system 10 has three modes, that is, a normal observationmode, a special observation mode, and a score display mode. In thenormal observation mode, the observation target is illuminated withnormal light and the image of the observation target is picked up,whereby a normal image having natural color tones is displayed on thedisplay 18. In the special observation mode, the observation target isilluminated with special light having a wavelength range different fromthe wavelength range of normal light and the image of the observationtarget is picked up, whereby a special image in which a specificstructure is enhanced is displayed on the display 18. In the scoredisplay mode, a score related to the determination of the severity orthe stage of the disease of the observation target is decided on thebasis of the endoscopic image consisting of the normal image or thespecial image, whereby, for example, the decided score, the change overtime of the score, and/or the determination result is displayed on thedisplay 18.

The severity of the disease is the degree of prognosis obtained bytreatment, and is classified into three categories, that is, mild,moderate, and severe, for example, in a case where the disease isulcerative colitis. The stage of the disease is classified into twocategories, that is, an active stage and a remission stage in a casewhere the disease is ulcerative colitis. Therefore, the determinationresult of the severity is any one of mild, moderate, or severe, and thedetermination result of the stage is any one of the active stage or theremission stage, or any one of remission or non-remission. In addition,the score is display with which the severity or the stage of the diseaseof the observation target can be recognized, and is a numerical value, asentence, or the like. In the present embodiment, a case where an imageprocessing device determines the remission or the non-remission ofulcerative colitis will be described.

The processor device 16 to which the endoscope 12 is connected is animage processing device that executes the score display mode. The imageprocessing device comprises a processor. In the image processing device,programs related to an image signal acquisition unit 51, a DSP 52, anoise reduction unit 53, a signal processing unit 55, a video signalgeneration unit 56, and the like are incorporated in a memory. Theprograms are operated by a control unit (not shown) formed of aprocessor, whereby functions such as the image signal acquisition unit51, the DSP 52, the noise reduction unit 53, the signal processing unit55, and the video signal generation unit 56 are realized. The scoredisplay mode may be executed in another configuration. For example, thefunctions of the image processing device may be provided in an externalimage processing system separated from the endoscope system 10, anexternal image processing device may receive the endoscopic image toexecute the score display mode, and the execution result may bedisplayed on an external display connected to the external imageprocessing system.

The processor device 16 is electrically connected to the display 18 andthe console 19. The display 18 outputs and displays, for example, animage of the observation target, a score, a change over time of thescore, a determination result, and/or information incidental to theimage of the observation target. The console 19 functions as a userinterface that receives an input operation, such as function settings.An external recording unit (not shown) that records images, imageinformation, or the like may be connected to the processor device 16.

In FIG. 3 , the light source device 14 emits illumination light withwhich the observation target is irradiated. The light source device 14comprises a light source unit 20 and a light source control unit 21 thatcontrols the light source unit 20. The light source unit 20 is formedof, for example, a semiconductor light source, such as a multi-colorlight emitting diode (LED), a combination of a laser diode and aphosphor, or a halogen light source, such as a xenon lamp. In addition,the light source unit 20 includes, for example, an optical filter thatis used to adjust the wavelength range of light emitted by the LED orthe like. The light source control unit 21 turns on/off each LED or thelike or adjusts the drive current and drive voltage of each LED or thelike, thereby controlling the amount of illumination light. Further, thelight source control unit 21 controls the wavelength range of theillumination light by changing the optical filter or the like.

As shown in FIG. 4 , in the present embodiment, the light source unit 20includes four-color LEDs, that is, a violet light emitting diode (V-LED)20 a, a blue light emitting diode (B-LED) 20 b, a green light emittingdiode (G-LED) 20 c, and a red light emitting diode (R-LED) 20 d.

As shown in FIG. 5 , the V-LED 20 a generates violet light V of whichthe central wavelength is 405±10 nm and the wavelength range is 380 to420 nm. The B-LED 20 b generates blue light B of which the centralwavelength is 460±10 nm and the wavelength range is 420 to 500 nm. TheG-LED 20 c generates green light G of which the wavelength range is 480to 600 nm. The R-LED 20 d generates red light R of which the centralwavelength is 620 to 630 nm and the wavelength range is 600 to 650 nm.Violet light V is short-wavelength light that is used to detect asuperficial blood vessel dense part, an intramucosal hemorrhage part, oran extramucosal hemorrhage part, which is used in the score displaymode, and preferably includes a central wavelength or a peak wavelengthof 410 nm. Therefore, it is preferable that the endoscopic image used inthe score display mode is an image obtained by picking up an image ofthe observation target illuminated with violet light V.

The light source control unit 21 controls the V-LED 20 a, the B-LED 20b, the G-LED 20 c, and the R-LED 20 d. The light source control unit 21controls the respective LEDs 20 a to 20 d so that normal light of whichthe light intensity ratios between violet light V, blue light B, greenlight G, and red light R are Vc:Bc:Gc:Rc is emitted in the normalobservation mode.

The light source control unit 21 controls the respective LEDs 20 a to 20d so that special light of which the light intensity ratios betweenviolet light V as short-wavelength light, blue light B, green light G,and red light R are Vs:Bs:Gs:Rs is emitted in the special observationmode or the score display mode. It is preferable that special lightenhances superficial blood vessels and the like. Therefore, it ispreferable to make the light intensity of violet light V larger than thelight intensity of blue light B, as the light intensity ratiosVs:Bs:Gs:Rs of the special light. For example, as shown in FIG. 6 , aratio of the light intensity Vs of violet light V to the light intensityBs of blue light B is set to “4:1”. Alternatively, as shown in FIG. 7 ,the light intensity ratios between violet light V, blue light B, greenlight G, and red light R are set to 1:0:0:0 and only violet light V asshort-wavelength light may be emitted, for the special light.

It should be noted that the light intensity ratios include a case wherethe ratio of at least one semiconductor light source is 0 (zero), in thepresent specification. Therefore, the light intensity ratios include acase where any one or two or more of the semiconductor light sources arenot turned on. The light source unit 20 is regarded to have lightintensity ratios, for example, even in a case where only onesemiconductor light source is turned on and the other threesemiconductor light sources are not turned on as in a case where thelight intensity ratios between violet light V, blue light B, green lightG, and red light R are 1:0:0:0.

Light emitted from each of the LEDs 20 a to 20 d is incident on a lightguide 41 through an optical path coupling unit (not shown) that isformed of a mirror, a lens, and the like. The light guide 41 isincorporated in the endoscope 12 and a universal cord (a cord thatconnects the endoscope 12 to the light source device 14 and theprocessor device 16). The light guide 41 propagates light from theoptical path coupling unit to the distal end portion 12 d of theendoscope 12.

The distal end portion 12 d of the endoscope 12 is provided with anillumination optical system 30 a and an image pickup optical system 30b. The illumination optical system 30 a has an illumination lens 42, andthe observation target is irradiated with illumination light propagatedby the light guide 41, through the illumination lens 42. The imagepickup optical system 30 b has an objective lens 43, a zoom lens 44, andan image pickup sensor 45. Various types of light, such as lightreflected from the observation target, scattered light, andfluorescence, are incident on the image pickup sensor 45 through theobjective lens 43 and the zoom lens 44. With this, the image of theobservation target is formed on the image pickup sensor 45. The zoomlens 44 freely moves between the telephoto end and the wide end with theoperation of the zoom operation portion 12 h, and magnifies and reducesthe observation target of which the image is formed on the image pickupsensor 45.

The image pickup sensor 45 is a color image pickup sensor provided withany one of a red color (R) filter, a green color (G) filter, or a bluecolor (B) filter for each pixel, and picks up the image of theobservation target to output image signals of respective RGB colors. Acharge coupled device (CCD) image pickup sensor or a complementarymetal-oxide semiconductor (CMOS) image pickup sensor can be used as theimage pickup sensor 45. Further, a complementary color image pickupsensor provided with color filters of complementary colors, that is,cyan (C), magenta (M), yellow (Y), and green (G) may be used instead ofthe color image pickup sensor 45 provided with color filters of theprimary colors. In a case where the complementary color image pickupsensor is used, the image signals of four colors of CMYG are output.Therefore, the same RGB image signals as those of the image pickupsensor 45 can be obtained by converting the image signals of the fourcolors of CMYG into the image signals of the three colors of RGB throughthe complementary color-primary color conversion. Alternatively, amonochrome image pickup sensor that is not provided with color filtersmay be used instead of the image pickup sensor 45.

The image pickup sensor 45 is driven and controlled by the image pickupcontrol unit (not shown). The control performed by the image pickupcontrol unit differs depending on the respective modes. In the normalobservation mode or the score display mode, the image pickup controlunit controls the image pickup sensor 45 to pick up the image of theobservation target illuminated with normal light. With this, Bc imagesignals are output from the B pixels of the image pickup sensor 45, Gcimage signals are output from the G pixels thereof, and Rc image signalsare output from the R pixels thereof. In the special observation mode orthe score display mode, the image pickup control unit controls the imagepickup sensor 45 to pick up the image of the observation targetilluminated with special light. With this, Bs image signals are outputfrom the B pixels of the image pickup sensor 45, Gs image signals areoutput from the G pixels thereof, and Rs image signals are output fromthe R pixels thereof.

A correlated double sampling/automatic gain control (CDS/AGC) circuit 46performs correlated double sampling (CDS) or automatic gain control(AGC) on analog image signals that are obtained from the image pickupsensor 45. The image signals that have been passed through the CDS/AGCcircuit 46 are converted into digital image signals by an analog/digital(A/D) converter 48. The digital image signals that have been subjectedto A/D conversion are input to the processor device 16.

The processor device 16 comprises the image signal acquisition unit 51,the digital signal processor (DSP) 52, the noise reduction unit 53, amemory 54, the signal processing unit 55, and the video signalgeneration unit 56. The signal processing unit 55 comprises a normalimage generation unit 61, a special image generation unit 62, and ascore processing unit 63.

The image signal acquisition unit 51 acquires the digital image signalsof an endoscopic image, which are input from the endoscope 12. Theacquired image signals are transmitted to the DSP 52. The DSP 52performs various types of signal processing, such as defect correctionprocessing, offset processing, gain correction processing, linear matrixprocessing, gamma conversion processing, demosaicing processing, and YCconversion processing, on the received image signals. In the defectcorrection processing, signals of defective pixels of the image pickupsensor 45 are corrected. In the offset processing, dark currentcomponents are removed from the image signals that have been subjectedto the defect correction processing, and an accurate zero level is set.In the gain correction processing, the image signals of each color,which have been subjected to the offset processing, are multiplied by aspecific gain, whereby the signal level of each image signal isadjusted.

The linear matrix processing for improving color reproducibility isperformed on the image signals of each color, which have been subjectedto the gain correction processing. After that, the brightness or chromasaturation of each image signal is adjusted through the gamma conversionprocessing. The demosaicing processing (also referred to as equalizationprocessing or demosaicing) is performed on the image signals that havebeen subjected to the gamma conversion processing, and signals oflacking color in each pixel are generated by interpolation. All thepixels are made to have signals of the respective RGB colors through thedemosaicing processing. The DSP 52 performs the YC conversion processingon each image signal that has been subjected to the demosaicingprocessing, and outputs brightness signals Y, color difference signalsCb, and color difference signals Cr to the noise reduction unit 53.

The noise reduction unit 53 performs noise reduction processing, whichis performed through, for example, a moving average method or medianfiltering method, on the image signals that have been subjected to thedemosaicing processing and the like by the DSP 52. The image signal withreduced noise is stored in the memory 54.

The signal processing unit 55 acquires the image signals that have beensubjected to noise reduction, from the memory 54. Then, signalprocessing, such as color conversion processing, hue enhancementprocessing, and structure enhancement processing, is performed asnecessary on the acquired image signals, and a color endoscopic image inwhich the observation target is imaged is generated. The colorconversion processing is processing for performing color conversionthrough 3×3 matrix processing, gradation transformation processing,three-dimensional lookup table (LUT) processing, and the like on theimage signals. The hue enhancement processing is performed on the imagesignals that have been subjected to the color conversion processing. Thestructure enhancement processing is processing for enhancing, forexample, a specific tissue or structure included in the observationtarget, such as blood vessels or pit patterns, and is performed on theimage signals that have been subjected to the hue enhancementprocessing.

The signal processing unit 55 comprises the normal image generation unit61, the special image generation unit 62, and the score processing unit63. The signal processing unit 55 sets a destination to which the imagesignals from the noise reduction unit 53 are transmitted to any one ofthe normal image generation unit 61, the special image generation unit62, or the score processing unit 63, in response to the set mode.Specifically, the image signals are input to the normal image generationunit 61, for example, in a case where the normal observation mode isset. The image signals are input to the special image generation unit 62in a case where the special observation mode is set. The image signalsare input to the score processing unit 63 in a case where the scoredisplay mode is set.

The normal image generation unit 61 performs image processing for anormal image on the input Rc image signals, Gc image signals, and Bcimage signals corresponding to one frame. The image processing for anormal image includes color conversion processing, such as 3×3 matrixprocessing, gradation transformation processing, and three-dimensionallook up table (LUT) processing, hue enhancement processing, andstructure enhancement processing, such as spatial frequency enhancement.The Rc image signals, the Gc image signals, and the Bc image signalsthat have been subjected to the image processing for a normal image areinput to the video signal generation unit 56 as a normal image.

The special image generation unit 62 performs image processing for aspecial image on the input Rs image signals, Gs image signals, and Bsimage signals corresponding to one frame. The image processing for aspecial image includes color conversion processing, such as 3×3 matrixprocessing, gradation transformation processing, and three-dimensionalLUT processing, hue enhancement processing, and structure enhancementprocessing, such as spatial frequency enhancement. The Rs image signals,the Gs image signals, and the Bs image signals that have been subjectedto the image processing for a special image are input to the videosignal generation unit 56 as a special image.

Since the endoscopic image generated by the signal processing unit 55 isa normal observation image in a case where the observation mode is thenormal observation mode, and is a special observation image in a casewhere the observation mode is the special observation mode, the contentsof the color conversion processing, the hue enhancement processing, andthe structure enhancement processing differ depending on the observationmodes. In the normal observation mode, the signal processing unit 55generates the normal observation image by performing the various typesof signal processing for making the observation target have naturalcolor tones. In the special observation mode, the signal processing unit55 generates the special observation image by performing the abovevarious types of signal processing for enhancing at least the bloodvessels of the observation target. In the special observation imagegenerated by the signal processing unit 55, blood vessels (so-calledsuperficial blood vessels) or blood located at a relatively shallowposition in the observation target with respect to the surface of themucous membrane have magenta-based color (for example, brown color), andblood vessels located at a relatively deep position in the observationtarget with respect to the surface of the mucous membrane (so-calledmedium-deep blood vessels) have cyan-based color (for example, greencolor). Therefore, the blood vessels or hemorrhage (blood) of theobservation target is enhanced by a difference in color with respect tothe mucous membrane represented by pink-based color.

The video signal generation unit 56 converts the normal image and thespecial image that are output from the signal processing unit 55, afinal score decided by the score processing unit 63, or the like intovideo signals allowing full-color display on the display 18. The videosignals that have been converted are input to the display 18. With this,the normal image, the special image, the final score, or the like isdisplayed on the display 18. Further, in a case where a still imageacquisition instruction (freeze instruction or release instruction) isinput by the operation of the still image acquisition instructionportion 12 f, the signal processing unit 55 stores the generatedendoscopic image in an image storage unit 75 (see FIG. 8 ) or a storage(not shown). The storage is an external storage device connected to theprocessor device 16 through a local area network (LAN) or the like, andis, for example, a file server of a system for filing an endoscopicimage, such as a picture archiving and communication system (PACS, seeFIG. 21 ), or a network attached storage (NAS).

The score processing unit 63 decides the final score, and performscontrol to display the final score and/or the change over time of thefinal score in real time on the display 18. As shown in FIG. 8 , thescore processing unit 63 comprises an image acquisition unit 71, a rawscore calculation unit 72, a final score decision unit 73, and a displaycontrol unit 76. Further, an unsuitable image discrimination unit 74,the image storage unit 75, and a site determination unit 77 may beprovided.

In the score display mode, the image acquisition unit 71 automaticallyacquires a plurality of endoscopic images obtained by picking up imagesof the observation target with the endoscope 12 at times different fromeach other. Although the endoscopic images include the normalobservation image and the special observation image, the imageacquisition unit 71 acquires a special observation image in which bloodvessels and the like are enhanced, in the present embodiment. The imageacquisition unit 71 may acquire the endoscopic image from the storage insome cases. The endoscopic image acquired by the image acquisition unit71 is sent to the raw score calculation unit 72, the unsuitable imagediscrimination unit 74, or the image storage unit 75.

The raw score calculation unit 72 calculates a raw score related to thedetermination of the severity or the stage of the disease of theobservation target on the basis of the plurality of endoscopic imagesacquired by the image acquisition unit 71. The raw score related to thedetermination of the severity or the stage of the disease of theobservation target is set to a numerical value with which the severityor the stage of the disease of the observation target included in theendoscopic image can be recognized. The raw score calculation unit 72can be provided with one or both of a first calculation unit 81 (seeFIG. 8 ) and a second calculation unit 82 (see FIG. 8 ).

It is preferable that the raw score calculation unit 72 calculates theraw score on the basis of a feature amount (first feature amount)obtained by analyzing the endoscopic image, through the firstcalculation unit 81. Examples of the feature amount include a featureamount related to the blood vessels, such as the number, thicknesses,lengths, number of branches, branch angles, distance between branchpoints, number of intersections, inclination, density, color, bloodconcentration, oxygen saturation, presence/absence of hemorrhage, areaof hemorrhage or flow rate of blood vessels, or a feature amount relatedto the color of the mucous membrane.

Examples of the feature amount of the raw score related to the severityor the stage of ulcerative colitis preferably include the amount relatedto the superficial blood vessel dense part, the intramucosal hemorrhagepart, and the extramucosal hemorrhage part included in the endoscopicimage. For the superficial blood vessel dense part, blood vessels of theobservation target are extracted from the endoscopic image by afrequency filter or the like through the image analysis on theendoscopic image, and a value obtained by counting the number of pixelsof a part where blood vessels are dense in the endoscopic image is setto the raw score. Similarly, for the intramucosal or extramucosalhemorrhage part, the intramucosal or extramucosal hemorrhage of theobservation target is extracted from the endoscopic image with a valueof a G value with respect to an R value proportional to the amount ofhemoglobin or the like, and a value obtained by counting the number ofpixels of the intramucosal or extramucosal hemorrhage part in theendoscopic image is set to the raw score.

The following method can also be used as a method of counting the numberof pixels of the superficial blood vessel dense part, the intramucosalhemorrhage part, and the extramucosal hemorrhage part included in theendoscopic image. First, the present inventor has found out thatulcerative colitis, which is a disease to be determined by the presentembodiment, changes the pattern of a blood vessel structure as theseverity is exacerbated, as shown in (A) to (E) of FIG. 9 . In a casewhere ulcerative colitis has remitted or ulcerative colitis has notoccurred, the patterns of superficial blood vessels 85 are regular ((A)of FIG. 9 ) or the regularity of the patterns of the superficial bloodvessels 85 is slightly disturbed ((B) of FIG. 9 ). On the other hand, ina case where ulcerative colitis has not remitted and the severity ofulcerative colitis is mild, the superficial blood vessels 85 are dense((C) of FIG. 9 ). Alternatively, in a case where ulcerative colitis hasnot remitted and the severity of ulcerative colitis is moderate,intramucosal hemorrhage 86 occurs ((D) of FIG. 9 ). Alternatively, in acase where ulcerative colitis has not remitted and the severity ofulcerative colitis is moderate to severe, extramucosal hemorrhage 87occurs ((E) of FIG. 9 ). The raw score calculation unit 72 can calculatethe raw score by using the above pattern change of the blood vesselstructure.

Here, the superficial blood vessel dense part refers to a state in whichsuperficial blood vessels meander and gather, and refers to a part inwhich many superficial blood vessels surround the crypt (see FIG. 10 )in terms of appearance on the image. The intramucosal hemorrhage refersto hemorrhage within the mucous membrane (see FIG. 10 ) and requires tobe discriminated from hemorrhage into the inner cavity. The intramucosalhemorrhage refers to hemorrhage in the mucous membrane, not the innercavity (the lumen, a hole having plicae) in terms of appearance on theimage. The extramucosal hemorrhage refers to a small amount of bloodthat flows into the lumen, blood that oozes out of the lumen or themucous membrane located in front of the endoscope even after the insideof the lumen has been washed and that can be visually recognized, orintraluminal blood with bleeding on a hemorrhagic mucous membrane.

The raw score calculation unit 72 classifies the superficial bloodvessel dense part, the intramucosal hemorrhage part, or the extramucosalhemorrhage part according to the frequency characteristic and thebrightness value obtained from the special observation image.Specifically, the superficial blood vessel dense part, the intramucosalhemorrhage part, or the extramucosal hemorrhage part are classified asshown in FIG. 11 . The denseness of superficial blood vessels isrepresented by a low brightness value and a high frequencycharacteristic. The intramucosal hemorrhage is represented by a mediumbrightness value and a medium frequency characteristic. The extramucosalhemorrhage is represented by a low brightness value and a low frequencycharacteristic. In a case where various structures of the specialobservation image are represented by brightness values and frequencycharacteristics, a blurred dark part, an endoscope shadow (a shadow thatcan be generated at the central part of an endoscopic image in a casewhere the distal end portion 12 d of the endoscope is moved along thelumen), or the like of the special observation image is also included,in addition to the above-described three, that is, the superficial bloodvessel dense part, the intramucosal hemorrhage part, and theextramucosal hemorrhage part. The denseness of superficial bloodvessels, the intramucosal hemorrhage, and the extramucosal hemorrhagethat are necessary for the determination of the remission or thenon-remission of ulcerative colitis are extracted from the superficialblood vessel dense part, the intramucosal hemorrhage part, or theextramucosal hemorrhage part by using the above-describedclassification.

For the spatial frequency, the spatial frequency component distributionis calculated by applying Laplacian filter to the special observationimage. For example, in a case where the standard deviation of thefrequencies of nine pixels that are disposed near a part including aspecific pixel is a constant value or less, the specific pixel isdefined as a pixel belonging to a low frequency region on the basis ofthe spatial frequency component distribution. A high frequency region isextracted by Hessian analysis for the spatial frequency componentdistribution. A medium frequency region is a medium frequency regioncorresponding to a part in which the low frequency region and the highfrequency region are excluded from the special observation image. Inthis way, it is possible to calculate the number of pixels of thesuperficial blood vessel dense part, the intramucosal hemorrhage part,or the extramucosal hemorrhage part by classifying the pixels of thespecial observation image by using the spatial frequency and thebrightness value. As described above, it is preferable that the diseaseis ulcerative colitis because the determination of the severity or thestage of the disease can be satisfactorily performed with the aboveconfiguration.

The raw score calculated on the basis of a different first featureamount is used as a different type of raw score. The raw scorecalculation unit 72 may calculate one type of raw score or two or moretypes of raw scores.

The raw score calculation unit 72 calculates the raw score on the basisof the endoscopic image, through the second calculation unit 82. Thesecond calculation unit 82 comprises a trained first machine learningmodel. The trained first machine learning model is generated by an inputof the past endoscopic image associated with the raw score to a machinelearning model. That is, the trained first machine learning model is amachine learning model generated by causing the machine learning modelto learn to correctly output the associated raw score in response to aninput of the past endoscopic image to the machine learning model.Therefore, the “trained” includes the adjustment of various parametersin addition to the input of the past endoscopic image associated withthe score to the machine learning model. For example, a numerical valueobtained by quantifying the severity or the stage of the disease of theobservation target, or a feature amount may be used as the raw scoreassociated with the past endoscopic image. For example, the firstmachine learning model may be provided with two or more types of machinelearning models corresponding to the respective feature amounts, and maycalculate two or more types of raw scores.

The final score decision unit 73 decides the final score on the basis ofthe raw score. The raw score calculated by the raw score calculationunit 72 is an amount related to the determination of the severity or thestage of the disease, and is an indicator of how symptomatic the diseaseis or how advanced the disease is. Therefore, the method of deciding thefinal score may be adjusted according to the purpose of thedetermination and the like. For example, in a case where the degree ofthe severity or the progress of the stage is determined, the final scoreis set in terms of a large raw score value, a high severity, and anadvanced stage, and the severity or the stage is determined for a partwhere the disease is most exacerbated. In this case, examples of themethod of deciding the final score include a method of setting therespective threshold values in advance for three types of raw scores,that is, the number of pixels of the superficial blood vessel densepart, the number of pixels of the intramucosal hemorrhage part, and thenumber of pixels of the extramucosal hemorrhage part, to decide a rawscore exceeding the threshold value as the final score, or to decide araw score having a largest excess of the preset threshold value in termsof a high raw value, among the respective values of the three types ofraw scores, as the final score. On the other hand, for example, in acase of screening, the threshold value is set low.

The display control unit 76 performs control to display the final scoreand/or the change over time of the final score in real time on thedisplay 18. Examples of the method of displaying the final score includea method of displaying the numerical value of the final score value inreal time on the display 18, a method of displaying the numerical valueof the final score value in a graph format as the change over time ofthe final score with a graph of which the vertical axis is the finalscore value and the horizontal axis is the passage of time, or a methodof displaying the final score value and the preset threshold value byusing a message for giving notice of the determination result of theseverity or the stage of the disease. Further, the display control unit76 also performs control not to display the final score or the like. Forexample, in a case where the final score is smaller than the presetthreshold value, the final score is not displayed. It should be notedthat displaying in real time means displaying immediately, and does notmean displaying at exactly the same time.

In a case where the position to be imaged is changed and the observationtarget is changed, the obtained endoscopic image is also changed, andthe raw score calculated on the basis of the endoscopic image is alsochanged. However, the change in the raw score is not displayed as it ison the display 18, and a final score using the raw score is decided asthe final score because the score processing unit 63 configured asdescribed above decides the final score by using the raw score and thendisplays the decided final score on the display 18, so that a morestable score is displayed on the display 18 even in real time. Further,since control not to display the final score is performed depending onthe final score value, a more stable score is displayed in a case wherethe score is displayed. In addition, the change over time of the finalscore is displayed in real time, so that it is possible to grasp at aglance, for example, the change of the severity or the stage because ofthe change of the position to be imaged or how high the final score isat the position where the severity is highest or the stage is the mostadvanced in a case where the position to be imaged is changed and theobservation target is changed, in one observation. Therefore, the usercan grasp the part where the severity, the stage of the disease, or thelike in the course of the observation is the worst while concentratingon the observation. As described above, with the image processingdevice, the final score does not vary even in a case where theobservation target is finely changed, and more stable and robust scorecalculation and display of the final score or the like are performed.Accordingly, the image processing device contributes to the preventionof oversight of the lesion of the user who performs the observation orto the simpler observation.

The score processing unit 63 may be provided with the unsuitable imagediscrimination unit 74. The unsuitable image discrimination unit 74discriminates whether the endoscopic image is suitable or unsuitable forthe calculation of the raw score. Whether the endoscopic image issuitable or unsuitable is discriminated through the discrimination. Theendoscopic image may be unsuitable as an image for calculating the rawscore. An extreme numerical value may be calculated in a case where theraw score is calculated, for example, for reasons such as that theobservation target is blurred because imaging is performed while thedistal end portion 12 d of the endoscope is moving, that the observationtarget is out of focus because water droplets are attached, that manyblurred parts are generated because the observation target is diagonallypositioned, or that the observation target is hardly included becauseonly the distant view is imaged, in the endoscopic image. Suchendoscopic images in which there is a concern that inappropriate scoresmay be calculated are discriminated to be unsuitable for the calculationof the raw score. The score processing unit 63 can be provided with oneor both of a first discrimination unit 83 (see FIG. 8 ) and a seconddiscrimination unit 84 (see FIG. 8 ).

The first discrimination unit 83 discriminates whether the endoscopicimage is suitable or unsuitable for the calculation of the raw score, onthe basis of the feature amount (second feature amount) obtained byanalyzing the endoscopic image. It is preferable that an amount relatedto at least one selected from halation distribution, spatial frequencydistribution, brightness value distribution, shadow distribution, amagnification ratio indicator, and reflected light distribution ofillumination light emitted to the observation target of the endoscopicimage is used as the feature amount in this case.

In the endoscopic image, since a region where halation occurs or aregion of shadow generated due to the hood or the like of the endoscopeis an extremely bright or dark region, the endoscopic image having alarge number of these regions is unsuitable for the calculation of theraw score. An endoscopic image having a large number of extremely brightor dark regions due to the brightness value distribution or the shadowdistribution is also unsuitable for the calculation of the raw score. Anendoscopic image having a large number of regions where the image isblurred due to the spatial frequency distribution is unsuitable for thecalculation of the raw score because the image is blurred or out offocus. In addition, in a case where a magnification ratio is large whena magnification ratio indicator is referred to, the imaged appearance ofblood vessels is changed and the blood vessel density per unit area ischanged, for example, in a case where the superficial blood vessel densepart is calculated, as compared with a case where no magnificationoccurs. Therefore, a case where the raw score is calculated inconsideration of the magnification ratio is suitable, and a case wherethe magnification ratio is not considered is unsuitable for thecalculation of the raw score. Further, in a case where the observationtarget is changed to a different type because of the change of anobservation site or the like, the reflected light distribution ofillumination light emitted to the observation target may be changed. Anendoscopic image having a large number of extremely bright or darkregions due to the change of the reflected light distribution is alsounsuitable for the calculation of the raw score.

Further, the value calculated on the basis of a different second featureamount is used as a different type of second feature amount. The firstdiscrimination unit 83 may calculate one type or two or more types ofsecond feature amounts. The first discrimination unit 83 discriminateswhether the endoscopic image is suitable or unsuitable for thecalculation of the raw score by using one type or two or more types ofsecond feature amounts.

The unsuitable image discrimination unit 74 discriminates whether theendoscopic image is suitable or unsuitable for the calculation of theraw score on the basis of the endoscopic image, through the seconddiscrimination unit 84. The second discrimination unit 84 comprises atrained second machine learning model. The trained second machinelearning model is generated by an input of the past endoscopic imageassociated with whether the endoscopic image is suitable or unsuitablefor the calculation of the raw score, to a machine learning model. Thatis, the trained second machine learning model is a machine learningmodel generated by causing the machine learning model to learn tocorrectly output whether the endoscopic image is suitable or unsuitablefor the calculation of the raw score, in response to an input of thepast endoscopic image to the machine learning model. Therefore, the“trained” includes the adjustment of various parameters in addition tothe input of the past endoscopic image associated with whether theendoscopic image is suitable or unsuitable for the calculation of theraw score, to the machine learning model.

An endoscopic image discriminated to be suitable for the calculation ofthe raw score by the unsuitable image discrimination unit 74 is sent tothe raw score calculation unit 72, and the raw score is calculated. Theraw score is defined to be “uncalculated” for an endoscopic imagediscriminated to be unsuitable for the calculation of the raw score.That is, since the raw score is labeled as “uncalculated”, theendoscopic image for which the raw score is defined to be uncalculatedis distinguished from the endoscopic image for which the raw score hasnot been calculated yet.

It is preferable that the final score decision unit 73 decides the finalscore from the raw score that is calculated on the basis of theplurality of endoscopic images acquired in a predetermined period beforea point in time at which the final score is decided. The decision of thefinal score in this case will be described in detail with reference toFIG. 12 . In FIG. 12 , the flow from the observation start through theendoscope is shown in the upper part, and the appearance of the obtainedendoscopic image is shown in the lower part. Observation through theendoscope is started, and endoscopic image acquisition 101 is startedfrom observation start time S. The image signal acquisition unit 51automatically performs the endoscopic image acquisition 101 at apredetermined image acquisition interval a. As soon as the endoscopicimage acquisition 101 is performed, the raw score of the endoscopicimage is calculated. Unsuitable image discrimination may be performedprior to the calculation. In FIG. 12 , although the endoscopic imageacquisition 101 is indicated by a filled circle, only a part thereof ismarked in order to avoid complicating the figure. Further, also for theimage acquisition interval a, only a part thereof is marked in order toavoid complicating the figure. The final score is decided on the basisof the endoscopic image acquired in a period Δt before final scoredecision time t, which is the decision time of the final score. Theperiod Δt is a predetermined period and is set in advance.

A first endoscopic image acquired in the period Δt is an endoscopicimage 121 acquired at time t−Δt. The endoscopic image 121 includes anintramucosal hemorrhage part 126 and a blood vessel dense part 127. InFIG. 12 , although the intramucosal hemorrhage part 126 and the bloodvessel dense part 127 are shown by a diagonally hatched portion and across hatched portion, respectively, only a part thereof is marked inorder to avoid complicating the figure. The unsuitable imagediscrimination unit 74 uses a period B to discriminate whether theendoscopic image 121 obtained by the endoscopic image acquisition 101 attime t−Δt is suitable or unsuitable for the calculation of the rawscore, and obtains the discrimination result at the time of unsuitableimage discrimination 102. The discrimination result is “suitable”.

The raw score calculation unit 72 uses a period C to calculate the rawscore in the endoscopic image 121 after the unsuitable imagediscrimination 102, and obtains the calculation result at the time ofraw score calculation 103. The raw score calculation unit calculates theraw scores through the first calculation unit 81 for two types, that is,the number of pixels of the intramucosal and extramucosal hemorrhageparts (hereinafter, referred to as the number of pixels of a hemorrhagepart) and the number of pixels of the blood vessel dense part(hereinafter, referred to as the number of pixels of a dense part). Thecalculation results are “the number of pixels of the hemorrhage part:100” and “the number of pixels of the dense part: 70”.

Next, among the endoscopic images acquired in the period Δt, a secondraw score is calculated for an endoscopic image 122 acquired at timet−Δt+4a. The endoscopic image 122 includes the intramucosal hemorrhagepart 126, the blood vessel dense part 127, and an extramucosalhemorrhage part 128. In FIG. 12 , although the extramucosal hemorrhagepart 128 is indicated by a filled circle, only a part thereof is markedin order to avoid complicating the figure. The unsuitable imagediscrimination unit 74 uses the period B to discriminate whether theendoscopic image 122 obtained by the endoscopic image acquisition 101 attime t−Δt+4a is suitable or unsuitable for the calculation of the rawscore, and obtains the discrimination result at the time of unsuitableimage discrimination 104. The discrimination result is “suitable”.

The raw score calculation unit 72 uses the period C to calculate the rawscore in the endoscopic image 122 after the unsuitable imagediscrimination 104, and obtains the calculation result at the time ofraw score calculation 105. The raw score calculation unit calculates theraw scores through the first calculation unit 81 for two types, that is,the number of pixels of the hemorrhage part and the number of pixels ofthe dense part. The calculation results are “the number of pixels of thehemorrhage part: 120” and “the number of pixels of the dense part: 90”.

Next, among the endoscopic images acquired in the period Δt, a third rawscore is calculated for an endoscopic image 123 acquired at timet−Δt+8a. Since the endoscopic image 123 is acquired while the endoscopeis moving, the endoscopic image 123 is an unclear image 129 in which theobservation target is blurred, and the observation target cannot bediscriminated. The unsuitable image discrimination unit 74 uses theperiod B to discriminate whether the endoscopic image 123 obtained bythe endoscopic image acquisition 101 at time t−Δt+8a is suitable orunsuitable for the calculation of the raw score, and obtains thediscrimination result at the time of unsuitable image discrimination106. The discrimination result is “unsuitable”.

The raw score calculation unit 72 receives that the result of theunsuitable image discrimination is “unsuitable” after the unsuitableimage discrimination 106, and does not calculate the raw score. That is,the calculation result of the raw score is defined to be “uncalculated”.

Next, among the endoscopic images acquired in the period Δt, a fourthraw score is calculated for an endoscopic image 124 acquired at timet−Δt+12a. The endoscopic image 124 includes the intramucosal hemorrhagepart 126, the blood vessel dense part 127, and the extramucosalhemorrhage part 128. The unsuitable image discrimination unit 74 usesthe period B to discriminate whether the endoscopic image 124 obtainedin the endoscopic image acquisition 101 at time t−Δt+12a is suitable orunsuitable for the calculation of the raw score, and obtains thediscrimination result at the time of unsuitable image discrimination108. The discrimination result is “suitable”.

The raw score calculation unit 72 uses the period C to calculate the rawscore in the endoscopic image 124 after the unsuitable imagediscrimination 108, and obtains the calculation result at the time ofraw score calculation 109. The raw score calculation unit calculates theraw scores through the first calculation unit 81 for two types, that is,the number of pixels of the hemorrhage part and the number of pixels ofthe dense part. The calculation results are “the number of pixels of thehemorrhage part: 140” and “the number of pixels of the dense part: 140”.

In this way, the endoscopic images acquired in the period Δt are thefour endoscopic images 121, 122, 123, and 124 in relation to the imageacquisition interval a, the period of the unsuitable imagediscrimination performed by the unsuitable image discrimination unit 74,the period of the raw score calculation performed by the raw scorecalculation unit 72. It is preferable that the final score decision unit73 decides the final score on the basis of the raw scores except for theraw score to be “uncalculated”, among these endoscopic images.Therefore, the final score decision unit 73 decides the final score fromthe respective raw scores calculated on the basis of the threeendoscopic images 121, 122, and 124.

It is preferable that the final score decision unit 73 decides the finalscore by performing a moving average, or finite impulse response (FIR)filtering or infinite impulse response (IIR) filtering of the pluralityof raw scores. The moving average is preferably any one of a simplemoving average, a weighted moving average, an exponential movingaverage, or a triangular moving average. In a case where weighting isused in the moving average, a value decided so as to obtain a preferableresult can be set in advance and used according to the observation site,a difference in the conditions for acquiring the endoscopic image, andthe like. The moving average, or FIR filtering or IIR filtering is used,so that it is possible to decide the final score, for example, with lessinfluence of noise, such as raw scores of extremely distant values, in aplurality of raw scores. Accordingly, the final score is stably decided.

In a case where the final score is determined by using, for example, asimple moving average of the raw scores at the final score decision timet, the numbers of pixels of the hemorrhage part in the endoscopic images121, 122, and 124 are averaged when the raw score is for the number ofpixels of the hemorrhage part. That is, for the first type of the finalscore, the number of pixels 100 of the hemorrhage part of the endoscopicimage 121, the number of pixels 120 of the hemorrhage part of theendoscopic image 122, and the number of pixels 140 of the hemorrhagepart of the endoscopic image 124 are averaged to 120 as in Equation (1).Similarly, for the second type of the final score, the number of pixels70 of the dense part of the endoscopic image 121, the number of pixels90 of the dense part of the endoscopic image 122, and the number ofpixels 140 of the dense part of the endoscopic image 124 are averaged to100 as in Equation (2).

Final score (the number of pixels of the hemorrhagepart)=(100+120+140)/3=120   (1)

Final score (the number of pixels of the dense part)=(70+90+140)/3=100  (2)

The final score decision interval may be automatically decided, or thefinal score at the point in time as instructed may be decided by aninstruction. The instruction may be given, for example, at the point intime when the user acquires the still image through the still imageacquisition instruction portion 12 f (freeze button) as the final scoredecision time t. Therefore, in this case, the display control unit 76performs control to display the final score and/or the change over timeof the final score in a case where a still image acquisition instructionthrough the still image acquisition instruction portion 12 f is given.The observation target for which the user stores the still image oftenincludes a region of interest. The decision of the final score in such aregion and the display thereof on the display 18 contribute to a moreappropriate diagnosis of the user, which is preferable.

Next, the display control unit 76 performs control to display the finalscore and/or the change over time of the final score in real time on thedisplay 18. As a display control method, a method capable of stablydisplaying a decision result of the final score is preferable.Therefore, examples of the display control method include a method ofupdating the display of the final score or a method of displaying thechange over time of the final score, in a case where the display controlunit 76 displays a final score having the largest numerical value amongthe final scores from the observation start and a final score largerthan the previous numerical values is obtained.

It is preferable to display the change over time of the final score as agraph in a case of displaying the change over time of the final score.It is preferable that the change over time of the final score isdisplayed by at least one graph showing a relationship between the finalscore and the final score decision time, as the graph. In a case ofdisplaying the change over time of the final score by a graph, it ispreferable to set a threshold value in advance for the final score valueand display the final score on the graph only in a case where the finalscore value is the threshold value or more.

As shown in FIG. 13 , the display control unit 76 performs control as towhether or not to display the final score with the threshold value atthe final score decision time t described above. In a case where thereare two types of final scores, two types of graphs are displayed on thedisplay 18. In the final score, a threshold value T1 for the number ofpixels of the hemorrhage part is set to 800, and a threshold value T2for the number of pixels of the dense part is set to 40. Since the finalscore of the number of pixels of the hemorrhage part at the final scoredecision time t is 120, the final score is smaller than 800 of thethreshold value T1. Therefore, the number of pixels of the hemorrhagepart at the final score decision time t is not plotted on a graph 131 ofthe number of pixels of the hemorrhage part, which is the final score.Further, since the final score of the number of pixels of the dense partat the final score decision time t is 100, the final score is largerthan 40 of the threshold value T2. Therefore, the number of pixels ofthe dense part at the final score decision time t is plotted with anauxiliary line 133 on a graph 132 of the number of pixels of the densepart, which is the final score, so that the numerical value can begrasped at a glance. In the graph 131 and the graph 132 of FIG. 13 , thethreshold value T1 or the threshold value T2 is indicated by diagonallines.

The display control unit 76 performs control to display the final scoreand/or the change over time of the final score in real time on thedisplay 18, so that the user can grasp at a glance which item thedisease is considered to be most exacerbated in while observing theobservation target. Further, the change over time is shown in a graph,so that it is possible to grasp the approximate position of a part wherethe disease is most exacerbated. Further, since only the item in whichthe disease is considered to be most exacerbated is displayed throughthe control of the display of the final score with the threshold value,the display of the final score can be made more stable.

Further, as the control of the display of the final score using thethreshold value, for example, the graph displayed on the display 18 maybe limited to one in a case where there are two or more types of finalscores. Specifically, for example, in a case where one out of the twotypes of final scores is the threshold value or more and the other isless than the threshold value, only the graph of the final score of thethreshold value or more is displayed. The types of parameters such asthe final score are carefully selected and displayed, so that thereliability or the stability of the displayed final score is improved.In addition, the user can grasp information for diagnosing the severityor the stage at a glance.

The site determination unit 77 (see FIG. 8 ) that determines the site ofthe observation target included in the endoscopic image by performingimage analysis on the endoscopic image may be provided, and the changeover time of the final score may be displayed by the graph showing arelationship between the final score, and the final score decision timeand the site. With this, the image analysis on the endoscopic image isautomatically performed, and the site name is displayed in associationwith the display of the change over time of the final score. As shown inFIG. 14 , for example, in a case of performing screening while pullingthe endoscope from the part of the cecum, which is positioned at thedeep part of the large intestine, in the observation of the largeintestine, a site name 134, such as “cecum, ascending colon, transversecolon, descending colon, sigmoid colon, and rectum”, is displayed. Inthis way, the site name 134 is displayed so that it is possible to graspthe part where the disease is exacerbated in the observation withoutomission, which is preferable.

The final score decision unit 73 may decide the final score on the basisof the raw score calculated immediately before or immediately after thepoint in time at which the final score is decided. The decision of thefinal score in these cases will be described in detail with reference toFIGS. 15 and 16 . As shown in FIG. 15 , in the method of deciding thefinal score in which the final score is decided on the basis of the rawscore calculated immediately before the point in time at which the finalscore is decided, the final score is decided on the basis of the rawscore calculated immediately before the point in time at which the finalscore is decided, instead of using the plurality of raw scores for thedecision. That is, the raw score calculated immediately before the pointin time at which the final score is decided is decided as the finalscore. The endoscopic image acquisition and the raw score acquisitionare the same as described above.

Since the raw scores calculated immediately before the final scoredecision time t are “the number of pixels of the hemorrhage part: 140”and “the number of pixels of the dense part: 140”, which are calculatedat the time of the raw score calculation 109 on the basis of theendoscopic image acquired by the endoscopic image acquisition at timet−Δt+12a, the final scores are “the number of pixels of the hemorrhagepart: 140, the number of pixels of the dense part: 140”. The display ofthe final score using the threshold value through the display controlunit 76 is the same as described above. Therefore, as shown in FIG. 15 ,the display 18 displays the graph 132 of the number of pixels of thedense part in which the number of pixels of the dense part is plotted at140 as the final score.

Alternatively, as shown in FIG. 16 , in the method of deciding the finalscore in which the final score is decided on the basis of the raw scorecalculated immediately after the point in time at which the final scoreis decided, the final score is decided on the basis of the raw scorecalculated immediately after the final score decision time t, instead ofusing the plurality of raw scores for the decision. That is, the rawscore calculated immediately after the final score decision time t isdecided as the final score. This is the raw score calculated for theendoscopic image acquired at the final score decision time t. Theendoscopic image acquisition and the raw score acquisition are the sameas described above.

Since the raw scores calculated immediately after the final scoredecision time t are “the number of pixels of the hemorrhage part: 160”and “the number of pixels of the dense part: 70”, which are calculatedat the time of the raw score calculation 111 on the basis of theendoscopic image acquired by the endoscopic image acquisition at time t,the final scores are “the number of pixels of the hemorrhage part: 160,the number of pixels of the dense part: 70”. The display of the finalscore using the threshold value through the display control unit 76 isthe same as described above. Therefore, as shown in FIG. 16 , thedisplay 18 displays the graph 132 of the number of pixels of the densepart in which the number of pixels of the dense part is plotted at 70 asthe final score.

The method of deciding the final score on the basis of the raw scorecalculated immediately before or immediately after the point in time atwhich the final score is decided is suitable in a case where it isdesired to grasp the final score of the observation target currentlybeing observed. Moreover, the latest final score can be stablydisplayed, which is suitable.

Further, in a case where the final score is decided at the point in timewhen the user acquires the still image through the still imageacquisition instruction portion 12 f, a method of deciding the finalscore on the basis of the raw score calculated immediately before orimmediately after the point in time at which the final score is decidedis combined with the above case so that it is possible to quickly andstably display the final score based on the observation site which is aregion of interest of the user and for which the still image acquisitioninstruction is given. Therefore, the display of the final score at thetiming when the user wants to obtain information regarding the diagnosisanswers the user's needs, which is preferable.

The final score decision unit 73 may decide whether to calculate thefinal score or define the final score to be uncalculated from the numberof the raw scores discriminated to be uncalculated by the unsuitableimage discrimination unit 74 and the number of calculated raw scores,among the raw scores based on the plurality of endoscopic imagesacquired in a predetermined period. Specifically, the final score isdefined to be “uncalculated” in a case where, among the raw scores basedon the plurality of endoscopic images acquired in the predeterminedperiod, the number of the raw scores except for the raw score to beuncalculated is a predetermined number or less, a case where the numberof the raw scores to be uncalculated is a predetermined number or more,a case where a ratio of the number of the raw scores to be uncalculatedto the number of the raw scores based on the plurality of endoscopicimages acquired in the predetermined period is a predetermined value ormore, or the like.

The decision of the final score in this case will be described in detailwith reference to FIGS. 17 and 18 . As shown in FIG. 17 , the endoscopicimage acquisition 101 is automatically performed at the imageacquisition interval a in the period Δt, which is a predeterminedperiod, and the endoscopic images are acquired at time t−Δt, timet−Δt+4a, time t−Δt+8a, and time t−Δt+12a. As soon as the endoscopicimage is acquired, the unsuitable image discrimination unit 74discriminates whether the endoscopic image acquired at each time afterthe period B is suitable or unsuitable for the calculation of the rawscore. The endoscopic images 122 and 123 acquired at time t−Δt+4a andtime t−Δt+8a, among the plurality of endoscopic images, are the unclearimages 129. Therefore, the raw score calculation unit 72 defines the rawscore to be “uncalculated” for each of the endoscopic images 122 and123.

In the present embodiment, in a case where the number of the raw scoresexcept for the raw score to be uncalculated is two or less, among theraw scores based on the plurality of endoscopic images acquired in thepredetermined period, the final score is defined to be “uncalculated”.Therefore, the case where the number of the plurality of endoscopicimages is four, the number of the raw scores to be uncalculated is two,and the number of the raw scores except for the raw scores to beuncalculated is two corresponds to “a case where the number of the rawscores except for the raw scores to be uncalculated is two or less”.Therefore, the final score at the final score decision time t is definedto be “uncalculated”. The display control unit 76 performs control notto display the final score to be uncalculated, on the display.Therefore, as shown in FIG. 18 , the display 18 does not plot the finalscore on both the graph 131 and the graph 132.

In the above case shown in FIG. 17 , in a case where a method ofdefining the final score to be “uncalculated” when the number of the rawscores to be uncalculated is a predetermined number or more is adopted,the final score is defined to be “uncalculated” because the number ofthe raw scores to be uncalculated is two, for example, in a case wherethe predetermined number is set to two. Similarly, in a case where amethod of defining the final score to be “uncalculated” when the ratioof the number of the raw scores to be uncalculated to the number of theraw scores based on the plurality of endoscopic images acquired in thepredetermined period is a predetermined value or more is adopted, forexample, when the predetermined value is set to 0.5, the final score isdefined to be “uncalculated” because the number of the endoscopic imagesacquired in the period Δt, that is, the number of the raw scoresacquired in the period Δt is four, the number of the raw scores to be“uncalculated”, among them, is two, and the above ratio is 2/4 (0.5).

As described above, whether to calculate the final score or define thefinal score to be uncalculated is decided from the number of the rawscores discriminated to be uncalculated by the unsuitable imagediscrimination unit 74 and the number of the calculated raw scores sothat the number or the ratio of raw scores, which are not preferable, isrestrained from being used, the final score is appropriately decided,and the score is stably displayed, which is preferable.

The final score may be displayed by a message for giving notice of thedetermination result of the severity or the stage of the disease of theobservation target. The severity or the stage is determined by the finalscore. For example, the display control unit 76 uses a threshold valuefor performing control as to whether or not to display the final score.That is, in the final score, the severity or the stage is determined bythe threshold value T1 of the number of pixels of the hemorrhage partand the threshold value T2 of the number of pixels of the dense part.With regard to the severity, a case where the number of pixels of thehemorrhage part is the threshold value T1 or more is determined to besevere, a case where the number of pixels of the hemorrhage part is lessthan the threshold value T1 and the number of pixels of the dense partis the threshold value T2 or more is determined to be moderate, or acase where the number of pixels of the hemorrhage part is less than thethreshold value T1 and the number of pixels of the dense part is lessthan the threshold value T2 is determined to be mild. With regard to thestage, a case where a determination is made to be severe when the numberof pixels of the hemorrhage part is the threshold value T1 or more and acase where a determination is made to be moderate when the number ofpixels of the hemorrhage part is less than the threshold value T1 andthe number of pixels of the dense part is the threshold value T2 or moreare determined as pathological non-remission, or a case where adetermination is made to be mild when the number of pixels of thehemorrhage part is less than the threshold value T1 and the number ofpixels of the dense part is less than the threshold value T2 isdetermined as the pathological remission.

As shown in FIG. 19 , a message for giving notice of the determinationresult of the severity or the stage is displayed, for example, bydisplaying a message 135 on a part of the display 18. In addition to“non-remission”, the message 135 is any one of message “severe”,“moderate”, or “mild” in the case of the severity, and any one ofmessage “remission” or “non-remission” in the case of the stage. Thefinal score is displayed as a message for giving notice of thedetermination result of the severity or the stage of the disease so thatthe user can obtain information for supporting the diagnosis of theseverity or the stage at a glance without interruption of theobservation, which is preferable.

Next, a series of flows of the score display mode will be described withreference to the flowchart shown in FIG. 20 . In a case where a mode isswitched to the score display mode, the observation target is irradiatedwith special light. The endoscope 12 picks up the image of theobservation target illuminated with special light (step ST110), therebyobtaining an endoscopic image, which is a special image captured at acertain point in time. In this flowchart, the final scores are set to beautomatically acquired at a predetermined interval. The imageacquisition unit 71 acquires the special image from the endoscope 12(step ST120).

The special image is sent to the unsuitable image discrimination unit74, and whether the special image is suitable or unsuitable for thecalculation of the raw score is discriminated. In a case of a specialimage discriminated to be suitable (YES in step ST130), the raw score iscalculated on the basis of the special image (step ST140). In a case ofa special image discriminated to be unsuitable (NO in step ST130), theraw score is defined to be “uncalculated” (step ST150).

The final score is decided from the calculation result of the raw score(step ST160). In a case where the final score is decided, the displaycontrol unit 76 controls the display of the final score (step ST170).The display displays the final score in a controlled display (stepST180). In a case of ending the observation (YES in step ST190), theobservation ends. In a case of not ending the observation (NO in stepST190), the process returns to the endoscopic image acquisition.

In the above embodiment, the present invention is applied to theendoscope system that performs processing on the endoscopic image, butthe present invention can also be applied to a medical image processingsystem that processes medical images other than the endoscopic image. Inthis case, the medical image processing system includes the imageprocessing device of the embodiment of the present invention. Further,the present invention can also be applied to a diagnosis support devicethat is used to provide diagnosis support to the user by using medicalimages. The present invention can also be applied to a medical servicesupport device that is used to support the medical service, such as adiagnostic report, by using medical images.

For example, as shown in FIG. 21 , a diagnosis support device 201 isused in combination with the modality, such as a medical imageprocessing system 202, and picture archiving and communication systems(PACS) 203. As shown in FIG. 22 , a medical service support device 210is connected to various examination apparatuses such as a first medicalimage processing system 211, a second medical image processing system212, . . . , and an Nth medical image processing system 213 through anynetwork 214. The medical service support device 210 receives medicalimages from the first medical image processing system 211, the secondmedical image processing system 212, . . . , the Nth medical imageprocessing system 213, and supports the medical service on the basis ofthe received medical images.

In the above embodiment, the hardware structures of the processing unitsthat execute various types of processing, such as the image signalacquisition unit 51, the DSP 52, the noise reduction unit 53, the signalprocessing unit 55, and the video signal generation unit 56, which areincluded in the processor device 16, are various processors to bedescribed below. The various processors include, for example, a centralprocessing unit (CPU), which is a general-purpose processor thatexecutes software (programs) to function as various processing units, aprogrammable logic device (PLD), such as a field programmable gate array(FPGA), which is a processor having a changeable circuit configurationafter manufacture, and a dedicated electrical circuit, which is aprocessor having a dedicated circuit configuration designed to executevarious types of processing.

One processing unit may be formed of one of these various processors, ormay be formed of a combination of two or more processors of the sametype or different types (for example, a plurality of FPGAs, or acombination of a CPU and an FPGA). Alternatively, a plurality ofprocessing units may be formed of one processor. A first example inwhich a plurality of processing units are formed of one processor is anaspect in which one or more CPUs and software are combined to constituteone processor and the processor functions as a plurality of processingunits, as typified by a computer such as a client or a server. A secondexample is an aspect in which a processor that realizes all of thefunctions of a system including the plurality of processing units withone integrated circuit (IC) chip is used, as typified by a system onchip (SoC) or the like. In this manner, various processing units areformed of one or more of the above-described various processors ashardware structures.

More specifically, the hardware structures of these various processorsare electrical circuitry in which circuit elements such as semiconductorelements are combined.

The present invention can also be implemented by the following anotherembodiment.

-   -   A processor device is provided. The processor device        -   acquires a plurality of endoscopic images obtained by            picking up images of an observation target at times            different from each other with an endoscope device,        -   calculates a raw score related to a determination of a            severity or a stage of a disease of the observation target,            on the basis of each of the endoscopic images,        -   decides a final score on the basis of the raw score, and        -   performs control to display the final score and/or a change            over time of the final score in real time on a display.

EXPLANATION OF REFERENCES

-   -   10: endoscope system    -   12: endoscope    -   12 a: insertion part    -   12 b: operation part    -   12 c: bendable portion    -   12 d: distal end portion    -   12 e: angle knob    -   12 f: still image acquisition instruction portion    -   12 g: mode changeover switch    -   12 h: zoom operation portion    -   14: light source device    -   16: processor device (image processing device)    -   18: display    -   19: console    -   20: light source unit    -   20 a: V-LED    -   20 b: B-LED    -   20 c: G-LED    -   20 d: R-LED    -   21: light source control unit    -   30 a: illumination optical system    -   30 b: image pickup optical system    -   41: light guide    -   42: illumination lens    -   43: objective lens    -   44: zoom lens    -   45: image pickup sensor    -   46: CDS/AGC circuit    -   47: A/D converter    -   51: image signal acquisition unit    -   52: DSP    -   53: noise reduction unit    -   54: memory    -   55: signal processing unit    -   56: video signal generation unit    -   61: normal image generation unit    -   62: special image generation unit    -   63: score processing unit    -   71: image acquisition unit    -   72: raw score calculation unit    -   73: final score decision unit    -   74: unsuitable image discrimination unit    -   75: image storage unit    -   76: display control unit    -   77: site determination unit    -   81: first calculation unit    -   82: second calculation unit    -   83: first discrimination unit    -   84: second discrimination unit    -   85: superficial blood vessels    -   86: intramucosal hemorrhage    -   87: extramucosal hemorrhage    -   101: endoscopic image acquisition    -   102, 104, 106, 108, 110, 111: unsuitable image discrimination    -   103, 105, 107, 109: raw score calculation    -   121 to 125: endoscopic image    -   126: intramucosal hemorrhage part    -   127: blood vessel dense part    -   128: extramucosal hemorrhage part    -   129: unclear image    -   131, 132: graph    -   133: auxiliary line    -   134: site name    -   135: message    -   201: diagnosis support device    -   202: medical image processing system    -   203: PACS    -   210: medical service support device    -   211: first medical image processing system    -   212: second medical image processing system    -   213: Nth medical image processing system    -   214: network    -   t: final score decision time    -   ST110 to ST190: step

What is claimed is:
 1. An image processing device comprising: aprocessor configured to: acquire a plurality of endoscopic imagesobtained by picking up images of an observation target at timesdifferent from each other with an endoscope; calculate a raw scorerelated to a severity or a stage of a disease of the observation target,on the basis of the plurality of the endoscopic images; decide a finalscore on the basis of the raw score; and perform control to display achange over time of the final score in real time on a display.
 2. Theimage processing device according to claim 1, wherein the processor isconfigured to calculate two or more types of raw scores different fromeach other.
 3. The image processing device according to claim 1, whereinthe processor is configured to calculate the raw score on the basis of afirst feature amount obtained by analyzing the endoscopic image.
 4. Theimage processing device according to claim 3, wherein the first featureamount is an amount related to a superficial blood vessel dense part, anintramucosal hemorrhage part, or an extramucosal hemorrhage partincluded in the endoscopic image.
 5. The image processing deviceaccording to claim 1, wherein the processor is configured to calculatethe raw score with a trained first machine learning model generated byan input of a past endoscopic image associated with the raw score to amachine learning model.
 6. The image processing device according toclaim 1, wherein the processor is configured to decide the final scorefrom the raw score that is calculated on the basis of a plurality of theendoscopic images acquired in a predetermined period before a point intime at which the final score is decided.
 7. The image processing deviceaccording to claim 6, wherein the processor is configured to decide thefinal score by performing a moving average, or FIR filtering or IIRfiltering of a plurality of the raw scores.
 8. The image processingdevice according to claim 1, wherein the processor is configured todecide the final score on the basis of the raw score calculatedimmediately before or immediately after a point in time at which thefinal score is decided.
 9. The image processing device according toclaim 1, wherein the processor is configured to: discriminate, for eachendoscopic image, whether the endoscopic image is suitable or unsuitablefor the calculation of the raw score; and define the raw score to beuncalculated for the endoscopic image discriminated to be unsuitable forthe calculation of the raw score.
 10. The image processing deviceaccording to claim 9, wherein the processor is configured todiscriminate whether the endoscopic image is suitable or unsuitable forthe calculation of the raw score, on the basis of a second featureamount of the endoscopic image.
 11. The image processing deviceaccording to claim 10, wherein the second feature amount is an amountrelated to at least one of halation distribution, spatial frequencydistribution, brightness value distribution, shadow distribution, amagnification ratio indicator, and reflected light distribution ofillumination light of the endoscopic image, the illumination light beingemitted to the observation target.
 12. The image processing deviceaccording to claim 10, wherein the processor is configured todiscriminate whether the endoscopic image is suitable or unsuitable forthe calculation of the raw score with a trained second machine learningmodel generated by an input of a past endoscopic image to a machinelearning model, the past endoscopic image being associated with whetherthe endoscopic image is suitable or unsuitable for the calculation ofthe raw score.
 13. The image processing device according to claim 9,wherein the processor is configured to decide the final score on thebasis of the raw score except for the raw score to be uncalculated. 14.The image processing device according to claim 9, wherein the processoris configured to decide the final score from the raw score that iscalculated on the basis of a plurality of the endoscopic images acquiredin a predetermined period before a point in time at which the finalscore is decided, and define the final score to be uncalculated in acase where the number of the raw scores except for the raw score to beuncalculated is a predetermined number or less, a case where the numberof the raw scores to be uncalculated is a predetermined number or more,or a case where a ratio of the number of the raw scores to beuncalculated to the number of the raw scores based on the plurality ofendoscopic images acquired in the predetermined period is apredetermined value or more, the raw scores being based on the pluralityof endoscopic images acquired in the predetermined period.
 15. The imageprocessing device according to claim 1, wherein the change over time ofthe final score is displayed by at least one graph showing arelationship between the final score and a decision time of the finalscore.
 16. The image processing device according to claim 1, wherein theprocessor is configured to determine a site of the observation targetincluded in the endoscopic image by performing image analysis on theendoscopic image, and the change over time of the final score isdisplayed by at least one graph showing a relationship between the finalscore, and a decision time of the final score and the site.
 17. Theimage processing device according to claim 1, wherein the processor isconfigured to: give an instruction to acquire a still image; and performcontrol to display the final score and/or the change over time of thefinal score in a case where the instruction is given.
 18. The imageprocessing device according to claim 1, wherein the disease isulcerative colitis.
 19. An endoscope system comprising the imageprocessing device according to claim 1 and an endoscope that picks up animage of an observation target, wherein the processor is configured toacquire a plurality of endoscopic images obtained by picking up theimages of an observation target at times different from each other,calculate a raw score related to a determination of a severity or astage of a disease of the observation target, on the basis of each ofthe endoscopic images, decide a final score on the basis of the rawscore, and perform control to display the final score or a change overtime of the final score in real time on a display.
 20. An imageprocessing method comprising: acquiring a plurality of endoscopic imagesobtained by picking up images of an observation target at timesdifferent from each other; calculating a raw score related to adetermination of a severity or a stage of a disease of the observationtarget, on the basis of each of the endoscopic images; deciding a finalscore on the basis of the raw score; and performing control to displaythe final score or a change over time of the final score in real time ona display.